Add Leader Election for Housekeeping in Sharded Cluster using MongoDB #529
Conversation
Codecov Report
@@ Coverage Diff @@
## main #529 +/- ##
==========================================
+ Coverage 51.50% 51.70% +0.20%
==========================================
Files 67 68 +1
Lines 6932 7048 +116
==========================================
+ Hits 3570 3644 +74
- Misses 2891 2930 +39
- Partials 471 474 +3
|
krapie
force-pushed
the
shard-cluster-housekeeping-with-mongodb
branch
5 times, most recently
from
b5a41c0
to
dad28f9
Compare
|
I personally think we can also use K8s CronJob Object to run housekeeping in cluster mode without using leader election for housekeeping. Using K8s CronJob will ease overhead for the server cluster to process leader elections. This might have a big impact when you have a huge cluster of servers. |
krapie
force-pushed
the
shard-cluster-housekeeping-with-mongodb
branch
from
dad28f9
to
23e9f1d
Compare
|
I forgot to re-request review on this PR 😄 Requesting it now. |
There was a problem hiding this comment.
Thanks for applying my requests.
I personally think we can also use K8s CronJob Object to run housekeeping in cluster mode without using leader election for housekeeping.
Using K8s CronJob will ease overhead for the server cluster to process leader elections. This might have a big impact when you have a huge cluster of servers.
Using CronJob seems to be a good and straightforward approach. If we choose to use it, we can run the cluster without a leader election.
However, we should consider how to provide private APIs that are meant to be exclusively called by internal components and should not be exposed publicly. The proposed idea doesn't seem to depend on k8s, as we only need to expose the private APIs.
https://kubernetes.io/docs/concepts/workloads/controllers/cron-jobs/#concurrency-policy
How would you prefer to proceed?
|
@hackerwins At first glance, I thought we could introduce some sort of housekeeping-only mode on the server or CLI and run it periodically using K8s CronJob (or maybe just run a housekeeping-only server pod) since housekeeping does not require any context to receive or provide (housekeeping fetches all projects and clients from the database, and results for housekeeping only get applied to the database). I will search for more context and information to get more ideas. |
hackerwins
force-pushed
the
main
branch
3 times, most recently
from
7892142
to
fdc2e1c
Compare
hackerwins
force-pushed
the
main
branch
2 times, most recently
from
fbc6098
to
7d65a57
Compare
What this PR does / why we need it:
Add leader election for housekeeping in sharded cluster using mongodb.
server/backend/electionpackage is added for leader election, withdatabaseimplementation.databaseimplementation uses mongoDB and TTL indexes to ensure that only one node can acquire housekeeping leader lease by preventing other nodes try to update the same document simultaneously on document expiry.yorkie-clusterHelm Chart is configured to enable--housekeeping-leader-electionto perform leader-only housekeeping in sharded cluster.hostnamevalue inserver/backend/backend.gois fixed. For more information, follow: Add user agent metrics PRWhich issue(s) this PR fixes:
Fixes #505
Special notes for your reviewer:
I have tested on K8s environment with
yorkie-clusterHelm chart with minikube, and it worked as expected.Does this PR introduce a user-facing change?:
Additional documentation:
Checklist: